{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# 使用微调的方法让模型对新闻分类更加准确\n",
    "\n",
    "在本实操手册中，开发者将使用ChatGLM3-6B base模型，对新闻分类数据集进行微调，并使用微调后的模型进行推理。\n",
    "本操作手册使用公开数据集，数据集中包含了新闻标题和新闻关键词，开发者需要根据这些信息，将新闻分类到15个类别中的一个。\n",
    "为了体现模型高效的学习能力，以及让用户更快的学习本手册，我们只使用了数据集中的一小部分数据，实际上，数据集中包含了超过40万条新闻数据。\n",
    "\n",
    "## 硬件要求\n",
    "本实践手册需要使用 FP16 精度的模型进行推理，因此，我们推荐使用至少 16GB 显存的 英伟达 GPU 来完成本实践手册。\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "首先，我们将原始的数据集格式转换为用于微调的`jsonl`格式，以方便进行微调。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# 路径可以根据实际情况修改\n",
    "input_file_path = 'data/toutiao_cat_data_example.txt'\n",
    "output_file_path = 'data/toutiao_cat_data_example.jsonl'\n",
    "\n",
    "# 提示词\n",
    "prompt_prefix = \"\"\"\n",
    "你是一个专业的新闻专家，请根据我提供的新闻信息，包括新闻标题，新闻关键词等信息，你要对每一行新闻类别进行分类并告诉我结果，不要返回其他信息和多于的文字，这些类别是:\n",
    "news_story\n",
    "news_culture\n",
    "news_sports\n",
    "news_finance\n",
    "news_house\n",
    "news_car\n",
    "news_edu\n",
    "news_tech\n",
    "news_military\n",
    "news_travel\n",
    "news_world\n",
    "stock\n",
    "news_agriculture\n",
    "news_game\n",
    "请选择其中一个类别并返回，你只要返回类别的名称，不要返回其他信息。让我们开始吧:\n",
    "\"\"\"\n",
    "\n",
    "# 分类代码和名称的映射\n",
    "category_map = {\n",
    "    \"100\": \"news_story\",\n",
    "    \"101\": \"news_culture\",\n",
    "    \"102\": \"news_entertainment\",\n",
    "    \"103\": \"news_sports\",\n",
    "    \"104\": \"news_finance\",\n",
    "    \"106\": \"news_house\",\n",
    "    \"107\": \"news_car\",\n",
    "    \"108\": \"news_edu\",\n",
    "    \"109\": \"news_tech\",\n",
    "    \"110\": \"news_military\",\n",
    "    \"112\": \"news_travel\",\n",
    "    \"113\": \"news_world\",\n",
    "    \"114\": \"stock\",\n",
    "    \"115\": \"news_agriculture\",\n",
    "    \"116\": \"news_game\"\n",
    "}\n",
    "\n",
    "def process_line(line):\n",
    "    # 分割每行数据\n",
    "    parts = line.strip().split('_!_')\n",
    "    if len(parts) != 5:\n",
    "        return None\n",
    "\n",
    "    # 提取所需字段\n",
    "    _, category_code, _, news_title, news_keywords = parts\n",
    "\n",
    "    # 构造 JSON 对象\n",
    "    news_title = news_title if news_title else \"无\"\n",
    "    news_keywords = news_keywords if news_keywords else \"无\"\n",
    "    json_obj = {\n",
    "        \"context\": prompt_prefix + f\"新闻标题: {news_title}\\n 新闻关键词: {news_keywords}\\n\",\n",
    "        \"target\": category_map.get(category_code, \"无\")\n",
    "    }\n",
    "    return json_obj\n",
    "\n",
    "def convert_to_jsonl(input_path, output_path):\n",
    "    with open(input_path, 'r', encoding='utf-8') as infile, \\\n",
    "         open(output_path, 'w', encoding='utf-8') as outfile:\n",
    "        for line in infile:\n",
    "            json_obj = process_line(line)\n",
    "            if json_obj:\n",
    "                json_line = json.dumps(json_obj, ensure_ascii=False)\n",
    "                outfile.write(json_line + '\\n')\n",
    "\n",
    "# 运行转换函数\n",
    "convert_to_jsonl(input_file_path, output_file_path)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:43:59.281779Z",
     "end_time": "2023-11-24T13:43:59.330679Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 使用没有微调的模型进行推理\n",
    "首先，我们先试用原本的模基座模型进行推理，并查看效果。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "PROMPT = \"\"\"\n",
    "你是一个专业的新闻专家，请根据我提供的新闻信息，包括新闻标题，新闻关键词等信息，你要对每一行新闻类别进行分类并告诉我结果，不要返回其他信息和多于的文字，这些类别是:\n",
    "news_story\n",
    "news_culture\n",
    "news_sports\n",
    "news_finance\n",
    "news_house\n",
    "news_car\n",
    "news_edu\n",
    "news_tech\n",
    "news_military\n",
    "news_travel\n",
    "news_world\n",
    "stock\n",
    "news_agriculture\n",
    "news_game\n",
    "请选择其中一个类别并返回，你只要返回类别的名称，不要返回其他信息。让我们开始吧:\n",
    "新闻标题：华为手机扛下敌人子弹，是什么技术让其在战争中大放异彩？\n",
    "新闻关键词: 华为手机\n",
    "\"\"\""
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:43:59.377486Z",
     "end_time": "2023-11-24T13:43:59.392276Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "data": {
      "text/plain": "Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "ed12c09cc7f443bab2f99b3ca7e99716"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from transformers import AutoModel, AutoTokenizer\n",
    "import  torch\n",
    "# 参数设置\n",
    "model_path = \"THUDM/chatglm3-6b-base\"\n",
    "tokenizer_path = model_path\n",
    "device = \"cuda\"\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, trust_remote_code=True)\n",
    "model = AutoModel.from_pretrained(model_path, load_in_8bit=False, trust_remote_code=True).to(device)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:43:59.377486Z",
     "end_time": "2023-11-24T13:44:11.180343Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "data": {
      "text/plain": "'news_house'"
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_new_tokens = 1024\n",
    "temperature = 0.4\n",
    "top_p = 0.9\n",
    "inputs = tokenizer(PROMPT, return_tensors=\"pt\").to(device)\n",
    "response = model.generate(input_ids=inputs[\"input_ids\"],max_new_tokens=max_new_tokens,temperature=temperature,top_p=top_p,do_sample=True)\n",
    "response = response[0, inputs[\"input_ids\"].shape[-1]:]\n",
    "origin_answer = tokenizer.decode(response, skip_special_tokens=True)\n",
    "origin_answer"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:44:11.183956Z",
     "end_time": "2023-11-24T13:44:12.313278Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [],
   "source": [
    "del model\n",
    "torch.cuda.empty_cache()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:44:12.313278Z",
     "end_time": "2023-11-24T13:44:12.463365Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "我们可以发现，模型并没有正确的对新闻进行分类。这可能是由模型训练阶段的数据导致的问题。那么，我们通过微调这个模型，能不能实现更好的效果呢？"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 使用官方的微调脚本进行微调\n",
    "\n",
    "在完成数据的切割之后，我们需要按照官方提供好的方式进行微调。我们使用的模型为`chatglm3-6b-base`基座模型，该模型相对于Chat模型，更容易上手微调，且更符合本章节的应用场景。\n",
    "\n",
    "我们将对应的参数设置好后，就可以直接执行下面的代码进行微调。该代码使用`Lora`方案进行微调，成本相较于全参微调大幅度降低。\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-11-24 13:44:13,160] torch.distributed.run: [WARNING] master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.\r\n",
      "11/24/2023 13:44:15 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False\r\n",
      "11/24/2023 13:44:15 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(\r\n",
      "_n_gpu=1,\r\n",
      "adafactor=False,\r\n",
      "adam_beta1=0.9,\r\n",
      "adam_beta2=0.999,\r\n",
      "adam_epsilon=1e-08,\r\n",
      "auto_find_batch_size=False,\r\n",
      "bf16=False,\r\n",
      "bf16_full_eval=False,\r\n",
      "data_seed=None,\r\n",
      "dataloader_drop_last=False,\r\n",
      "dataloader_num_workers=0,\r\n",
      "dataloader_pin_memory=True,\r\n",
      "ddp_backend=None,\r\n",
      "ddp_broadcast_buffers=None,\r\n",
      "ddp_bucket_cap_mb=None,\r\n",
      "ddp_find_unused_parameters=None,\r\n",
      "ddp_timeout=1800,\r\n",
      "debug=[],\r\n",
      "deepspeed=None,\r\n",
      "disable_tqdm=False,\r\n",
      "dispatch_batches=None,\r\n",
      "do_eval=False,\r\n",
      "do_predict=False,\r\n",
      "do_train=False,\r\n",
      "eval_accumulation_steps=None,\r\n",
      "eval_delay=0,\r\n",
      "eval_steps=None,\r\n",
      "evaluation_strategy=no,\r\n",
      "fp16=False,\r\n",
      "fp16_backend=auto,\r\n",
      "fp16_full_eval=False,\r\n",
      "fp16_opt_level=O1,\r\n",
      "fsdp=[],\r\n",
      "fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},\r\n",
      "fsdp_min_num_params=0,\r\n",
      "fsdp_transformer_layer_cls_to_wrap=None,\r\n",
      "full_determinism=False,\r\n",
      "generation_config=None,\r\n",
      "generation_max_length=None,\r\n",
      "generation_num_beams=None,\r\n",
      "gradient_accumulation_steps=2,\r\n",
      "gradient_checkpointing=False,\r\n",
      "gradient_checkpointing_kwargs=None,\r\n",
      "greater_is_better=None,\r\n",
      "group_by_length=False,\r\n",
      "half_precision_backend=auto,\r\n",
      "hub_always_push=False,\r\n",
      "hub_model_id=None,\r\n",
      "hub_private_repo=False,\r\n",
      "hub_strategy=every_save,\r\n",
      "hub_token=<HUB_TOKEN>,\r\n",
      "ignore_data_skip=False,\r\n",
      "include_inputs_for_metrics=False,\r\n",
      "include_tokens_per_second=False,\r\n",
      "jit_mode_eval=False,\r\n",
      "label_names=None,\r\n",
      "label_smoothing_factor=0.0,\r\n",
      "learning_rate=2e-05,\r\n",
      "length_column_name=length,\r\n",
      "load_best_model_at_end=False,\r\n",
      "local_rank=0,\r\n",
      "log_level=passive,\r\n",
      "log_level_replica=warning,\r\n",
      "log_on_each_node=True,\r\n",
      "logging_dir=output/news-20231124-134412-2e-05/runs/Nov24_13-44-15_iZwz90pbe3r8jeoaaobbf0Z,\r\n",
      "logging_first_step=False,\r\n",
      "logging_nan_inf_filter=True,\r\n",
      "logging_steps=1.0,\r\n",
      "logging_strategy=steps,\r\n",
      "lr_scheduler_type=linear,\r\n",
      "max_grad_norm=1.0,\r\n",
      "max_steps=300,\r\n",
      "metric_for_best_model=None,\r\n",
      "mp_parameters=,\r\n",
      "neftune_noise_alpha=None,\r\n",
      "no_cuda=False,\r\n",
      "num_train_epochs=3.0,\r\n",
      "optim=adamw_torch,\r\n",
      "optim_args=None,\r\n",
      "output_dir=output/news-20231124-134412-2e-05,\r\n",
      "overwrite_output_dir=False,\r\n",
      "past_index=-1,\r\n",
      "per_device_eval_batch_size=8,\r\n",
      "per_device_train_batch_size=1,\r\n",
      "predict_with_generate=False,\r\n",
      "prediction_loss_only=False,\r\n",
      "push_to_hub=False,\r\n",
      "push_to_hub_model_id=None,\r\n",
      "push_to_hub_organization=None,\r\n",
      "push_to_hub_token=<PUSH_TO_HUB_TOKEN>,\r\n",
      "ray_scope=last,\r\n",
      "remove_unused_columns=True,\r\n",
      "report_to=[],\r\n",
      "resume_from_checkpoint=None,\r\n",
      "run_name=output/news-20231124-134412-2e-05,\r\n",
      "save_on_each_node=False,\r\n",
      "save_safetensors=True,\r\n",
      "save_steps=100,\r\n",
      "save_strategy=steps,\r\n",
      "save_total_limit=None,\r\n",
      "seed=42,\r\n",
      "skip_memory_metrics=True,\r\n",
      "sortish_sampler=False,\r\n",
      "split_batches=False,\r\n",
      "tf32=None,\r\n",
      "torch_compile=False,\r\n",
      "torch_compile_backend=None,\r\n",
      "torch_compile_mode=None,\r\n",
      "torchdynamo=None,\r\n",
      "tpu_metrics_debug=False,\r\n",
      "tpu_num_cores=None,\r\n",
      "use_cpu=False,\r\n",
      "use_ipex=False,\r\n",
      "use_legacy_prediction_loop=False,\r\n",
      "use_mps_device=False,\r\n",
      "warmup_ratio=0.0,\r\n",
      "warmup_steps=0,\r\n",
      "weight_decay=0.0,\r\n",
      ")\r\n",
      "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file tokenizer.model\r\n",
      "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file added_tokens.json\r\n",
      "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file special_tokens_map.json\r\n",
      "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file tokenizer_config.json\r\n",
      "[INFO|tokenization_utils_base.py:2020] 2023-11-24 13:44:15,353 >> loading file tokenizer.json\r\n",
      "[INFO|configuration_utils.py:715] 2023-11-24 13:44:15,453 >> loading configuration file /Models/chatglm3-6b-base/config.json\r\n",
      "[INFO|configuration_utils.py:715] 2023-11-24 13:44:15,454 >> loading configuration file /Models/chatglm3-6b-base/config.json\r\n",
      "[INFO|configuration_utils.py:777] 2023-11-24 13:44:15,454 >> Model config ChatGLMConfig {\r\n",
      "  \"_name_or_path\": \"/Models/chatglm3-6b-base\",\r\n",
      "  \"add_bias_linear\": false,\r\n",
      "  \"add_qkv_bias\": true,\r\n",
      "  \"apply_query_key_layer_scaling\": true,\r\n",
      "  \"apply_residual_connection_post_layernorm\": false,\r\n",
      "  \"architectures\": [\r\n",
      "    \"ChatGLMModel\"\r\n",
      "  ],\r\n",
      "  \"attention_dropout\": 0.0,\r\n",
      "  \"attention_softmax_in_fp32\": true,\r\n",
      "  \"auto_map\": {\r\n",
      "    \"AutoConfig\": \"configuration_chatglm.ChatGLMConfig\",\r\n",
      "    \"AutoModel\": \"modeling_chatglm.ChatGLMForConditionalGeneration\",\r\n",
      "    \"AutoModelForCausalLM\": \"modeling_chatglm.ChatGLMForConditionalGeneration\",\r\n",
      "    \"AutoModelForSeq2SeqLM\": \"modeling_chatglm.ChatGLMForConditionalGeneration\",\r\n",
      "    \"AutoModelForSequenceClassification\": \"modeling_chatglm.ChatGLMForSequenceClassification\"\r\n",
      "  },\r\n",
      "  \"bias_dropout_fusion\": true,\r\n",
      "  \"classifier_dropout\": null,\r\n",
      "  \"eos_token_id\": 2,\r\n",
      "  \"ffn_hidden_size\": 13696,\r\n",
      "  \"fp32_residual_connection\": false,\r\n",
      "  \"hidden_dropout\": 0.0,\r\n",
      "  \"hidden_size\": 4096,\r\n",
      "  \"kv_channels\": 128,\r\n",
      "  \"layernorm_epsilon\": 1e-05,\r\n",
      "  \"model_type\": \"chatglm\",\r\n",
      "  \"multi_query_attention\": true,\r\n",
      "  \"multi_query_group_num\": 2,\r\n",
      "  \"num_attention_heads\": 32,\r\n",
      "  \"num_layers\": 28,\r\n",
      "  \"original_rope\": true,\r\n",
      "  \"pad_token_id\": 0,\r\n",
      "  \"padded_vocab_size\": 65024,\r\n",
      "  \"post_layer_norm\": true,\r\n",
      "  \"pre_seq_len\": null,\r\n",
      "  \"prefix_projection\": false,\r\n",
      "  \"quantization_bit\": 0,\r\n",
      "  \"rmsnorm\": true,\r\n",
      "  \"seq_length\": 32768,\r\n",
      "  \"tie_word_embeddings\": false,\r\n",
      "  \"torch_dtype\": \"float16\",\r\n",
      "  \"transformers_version\": \"4.35.2\",\r\n",
      "  \"use_cache\": true,\r\n",
      "  \"vocab_size\": 65024\r\n",
      "}\r\n",
      "\r\n",
      "[INFO|modeling_utils.py:3118] 2023-11-24 13:44:15,517 >> loading weights file /Models/chatglm3-6b-base/pytorch_model.bin.index.json\r\n",
      "[INFO|configuration_utils.py:791] 2023-11-24 13:44:15,517 >> Generate config GenerationConfig {\r\n",
      "  \"eos_token_id\": 2,\r\n",
      "  \"pad_token_id\": 0\r\n",
      "}\r\n",
      "\r\n",
      "Loading checkpoint shards: 100%|██████████████████| 7/7 [00:07<00:00,  1.02s/it]\r\n",
      "[INFO|modeling_utils.py:3950] 2023-11-24 13:44:22,704 >> All model checkpoint weights were used when initializing ChatGLMForConditionalGeneration.\r\n",
      "\r\n",
      "[INFO|modeling_utils.py:3958] 2023-11-24 13:44:22,704 >> All the weights of ChatGLMForConditionalGeneration were initialized from the model checkpoint at /Models/chatglm3-6b-base.\r\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use ChatGLMForConditionalGeneration for predictions without further training.\r\n",
      "[INFO|modeling_utils.py:3525] 2023-11-24 13:44:22,706 >> Generation config file not found, using a generation config created from the model config.\r\n",
      "Train dataset size: 4999\r\n",
      "Sanity Check >>>>>>>>>>>>>\r\n",
      "           '[gMASK]':  64790 ->   -100\r\n",
      "               'sop':  64792 ->   -100\r\n",
      "                  '':  30910 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "                 '你':  54622 ->   -100\r\n",
      "               '是一个':  32103 ->   -100\r\n",
      "               '专业的':  34917 ->   -100\r\n",
      "                '新闻':  31935 ->   -100\r\n",
      "                '专家':  32114 ->   -100\r\n",
      "                 '，':  31123 ->   -100\r\n",
      "                 '请':  55073 ->   -100\r\n",
      "                '根据':  31793 ->   -100\r\n",
      "                 '我':  54546 ->   -100\r\n",
      "               '提供的':  35100 ->   -100\r\n",
      "                '新闻':  31935 ->   -100\r\n",
      "                '信息':  31707 ->   -100\r\n",
      "                 '，':  31123 ->   -100\r\n",
      "                '包括':  31779 ->   -100\r\n",
      "                '新闻':  31935 ->   -100\r\n",
      "                '标题':  34490 ->   -100\r\n",
      "                 '，':  31123 ->   -100\r\n",
      "                '新闻':  31935 ->   -100\r\n",
      "               '关键词':  35075 ->   -100\r\n",
      "               '等信息':  46172 ->   -100\r\n",
      "                 '，':  31123 ->   -100\r\n",
      "                '你要':  34526 ->   -100\r\n",
      "                 '对':  54570 ->   -100\r\n",
      "                '每一':  32467 ->   -100\r\n",
      "                 '行':  54560 ->   -100\r\n",
      "                '新闻':  31935 ->   -100\r\n",
      "                '类别':  38724 ->   -100\r\n",
      "                '进行':  31636 ->   -100\r\n",
      "                '分类':  33328 ->   -100\r\n",
      "                 '并':  54724 ->   -100\r\n",
      "               '告诉我':  38953 ->   -100\r\n",
      "                '结果':  31951 ->   -100\r\n",
      "                 '，':  31123 ->   -100\r\n",
      "                '不要':  31844 ->   -100\r\n",
      "                '返回':  34891 ->   -100\r\n",
      "                '其他':  31722 ->   -100\r\n",
      "               '信息和':  52701 ->   -100\r\n",
      "                 '多':  54573 ->   -100\r\n",
      "                 '于':  54579 ->   -100\r\n",
      "               '的文字':  48746 ->   -100\r\n",
      "                 '，':  31123 ->   -100\r\n",
      "                '这些':  31704 ->   -100\r\n",
      "                '类别':  38724 ->   -100\r\n",
      "                 '是':  54532 ->   -100\r\n",
      "                 ':':  30954 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "             'story':  12553 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "           'culture':  27458 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "                 's':  30917 ->   -100\r\n",
      "             'ports':   3915 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "               'fin':   6242 ->   -100\r\n",
      "              'ance':    562 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "             'house':   4199 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "               'car':   6747 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "               'edu':   7473 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "              'tech':  12232 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "               'mil':  20477 ->   -100\r\n",
      "             'itary':   2733 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "               'tra':   7441 ->   -100\r\n",
      "               'vel':    609 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "             'world':   8515 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "             'stock':  14148 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "                'ag':    369 ->   -100\r\n",
      "               'ric':    995 ->   -100\r\n",
      "            'ulture':   4768 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   8480 ->   -100\r\n",
      "                 '_':  30962 ->   -100\r\n",
      "              'game':   8947 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "                 '请':  55073 ->   -100\r\n",
      "                '选择':  31768 ->   -100\r\n",
      "              '其中一个':  46753 ->   -100\r\n",
      "                '类别':  38724 ->   -100\r\n",
      "                 '并':  54724 ->   -100\r\n",
      "                '返回':  34891 ->   -100\r\n",
      "                 '，':  31123 ->   -100\r\n",
      "                 '你':  54622 ->   -100\r\n",
      "                '只要':  32100 ->   -100\r\n",
      "                '返回':  34891 ->   -100\r\n",
      "                 '类':  54931 ->   -100\r\n",
      "                '别的':  34752 ->   -100\r\n",
      "                '名称':  33624 ->   -100\r\n",
      "                 '，':  31123 ->   -100\r\n",
      "                '不要':  31844 ->   -100\r\n",
      "                '返回':  34891 ->   -100\r\n",
      "                '其他':  31722 ->   -100\r\n",
      "                '信息':  31707 ->   -100\r\n",
      "                 '。':  31155 ->   -100\r\n",
      "               '让我们':  32817 ->   -100\r\n",
      "                '开始':  31699 ->   -100\r\n",
      "                 '吧':  55370 ->   -100\r\n",
      "                 ':':  30954 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "                '新闻':  31935 ->   -100\r\n",
      "                '标题':  34490 ->   -100\r\n",
      "                 ':':  30954 ->   -100\r\n",
      "                  '':  30910 ->   -100\r\n",
      "                '京城':  46921 ->   -100\r\n",
      "                 '最':  54628 ->   -100\r\n",
      "                '值得':  32421 ->   -100\r\n",
      "                '你来':  52586 ->   -100\r\n",
      "                 '场':  54686 ->   -100\r\n",
      "                '文化':  31653 ->   -100\r\n",
      "                '之旅':  35383 ->   -100\r\n",
      "                 '的':  54530 ->   -100\r\n",
      "               '博物馆':  32964 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "                '新闻':  45302 ->   -100\r\n",
      "               '关键词':  35075 ->   -100\r\n",
      "                 ':':  30954 ->   -100\r\n",
      "                  '':  30910 ->   -100\r\n",
      "                '保利':  46340 ->   -100\r\n",
      "                '集团':  31839 ->   -100\r\n",
      "                 ',':  30932 ->   -100\r\n",
      "                 '马':  54988 ->   -100\r\n",
      "                 '未':  54933 ->   -100\r\n",
      "                 '都':  54606 ->   -100\r\n",
      "                 ',':  30932 ->   -100\r\n",
      "                '中国':  31626 ->   -100\r\n",
      "              '科学技术':  35587 ->   -100\r\n",
      "                 '馆':  55294 ->   -100\r\n",
      "                 ',':  30932 ->   -100\r\n",
      "               '博物馆':  32964 ->   -100\r\n",
      "                 ',':  30932 ->   -100\r\n",
      "               '新中国':  35873 ->   -100\r\n",
      "                '\\n':     13 ->   -100\r\n",
      "              'news':   2374 ->   2374\r\n",
      "                 '_':  30962 ->  30962\r\n",
      "           'culture':  27458 ->  27458\r\n",
      "                  '':      2 ->      2\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "                  '':      0 ->   -100\r\n",
      "<<<<<<<<<<<<< Sanity Check\r\n",
      "[INFO|trainer.py:544] 2023-11-24 13:44:24,838 >> max_steps is given, it will override any value given in num_train_epochs\r\n",
      "[INFO|trainer.py:1723] 2023-11-24 13:44:25,787 >> ***** Running training *****\r\n",
      "[INFO|trainer.py:1724] 2023-11-24 13:44:25,787 >>   Num examples = 4,999\r\n",
      "[INFO|trainer.py:1725] 2023-11-24 13:44:25,787 >>   Num Epochs = 1\r\n",
      "[INFO|trainer.py:1726] 2023-11-24 13:44:25,787 >>   Instantaneous batch size per device = 1\r\n",
      "[INFO|trainer.py:1729] 2023-11-24 13:44:25,787 >>   Total train batch size (w. parallel, distributed & accumulation) = 2\r\n",
      "[INFO|trainer.py:1730] 2023-11-24 13:44:25,787 >>   Gradient Accumulation steps = 2\r\n",
      "[INFO|trainer.py:1731] 2023-11-24 13:44:25,788 >>   Total optimization steps = 300\r\n",
      "[INFO|trainer.py:1732] 2023-11-24 13:44:25,788 >>   Number of trainable parameters = 1,949,696\r\n",
      "  0%|                                                   | 0/300 [00:00<?, ?it/s][W reducer.cpp:1346] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())\r\n",
      "{'loss': 0.6584, 'learning_rate': 1.9933333333333334e-05, 'epoch': 0.0}         \r\n",
      "{'loss': 1.1299, 'learning_rate': 1.9866666666666667e-05, 'epoch': 0.0}         \r\n",
      "{'loss': 1.2722, 'learning_rate': 1.98e-05, 'epoch': 0.0}                       \r\n",
      "{'loss': 0.5076, 'learning_rate': 1.9733333333333336e-05, 'epoch': 0.0}         \r\n",
      "{'loss': 1.5129, 'learning_rate': 1.9666666666666666e-05, 'epoch': 0.0}         \r\n",
      "{'loss': 1.7524, 'learning_rate': 1.9600000000000002e-05, 'epoch': 0.0}         \r\n",
      "{'loss': 0.7554, 'learning_rate': 1.9533333333333335e-05, 'epoch': 0.0}         \r\n",
      "{'loss': 0.834, 'learning_rate': 1.9466666666666668e-05, 'epoch': 0.0}          \r\n",
      "{'loss': 2.0967, 'learning_rate': 1.94e-05, 'epoch': 0.0}                       \r\n",
      "{'loss': 0.2806, 'learning_rate': 1.9333333333333333e-05, 'epoch': 0.0}         \r\n",
      "{'loss': 1.0696, 'learning_rate': 1.926666666666667e-05, 'epoch': 0.0}          \r\n",
      "{'loss': 0.5934, 'learning_rate': 1.9200000000000003e-05, 'epoch': 0.0}         \r\n",
      "{'loss': 0.8784, 'learning_rate': 1.9133333333333335e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 1.9795, 'learning_rate': 1.9066666666666668e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.953, 'learning_rate': 1.9e-05, 'epoch': 0.01}                        \r\n",
      "{'loss': 0.1516, 'learning_rate': 1.8933333333333334e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.7275, 'learning_rate': 1.886666666666667e-05, 'epoch': 0.01}         \r\n",
      "{'loss': 0.2672, 'learning_rate': 1.88e-05, 'epoch': 0.01}                      \r\n",
      "{'loss': 0.152, 'learning_rate': 1.8733333333333336e-05, 'epoch': 0.01}         \r\n",
      "{'loss': 0.3321, 'learning_rate': 1.866666666666667e-05, 'epoch': 0.01}         \r\n",
      "{'loss': 0.5358, 'learning_rate': 1.86e-05, 'epoch': 0.01}                      \r\n",
      "{'loss': 0.3262, 'learning_rate': 1.8533333333333334e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.3477, 'learning_rate': 1.8466666666666667e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.2667, 'learning_rate': 1.8400000000000003e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.3765, 'learning_rate': 1.8333333333333333e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.1945, 'learning_rate': 1.826666666666667e-05, 'epoch': 0.01}         \r\n",
      "{'loss': 0.2321, 'learning_rate': 1.8200000000000002e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.094, 'learning_rate': 1.8133333333333335e-05, 'epoch': 0.01}         \r\n",
      "{'loss': 0.2454, 'learning_rate': 1.8066666666666668e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.0523, 'learning_rate': 1.8e-05, 'epoch': 0.01}                       \r\n",
      "{'loss': 0.9747, 'learning_rate': 1.7933333333333333e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.7521, 'learning_rate': 1.7866666666666666e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.7373, 'learning_rate': 1.7800000000000002e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.2056, 'learning_rate': 1.7733333333333335e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.0107, 'learning_rate': 1.7666666666666668e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.0525, 'learning_rate': 1.76e-05, 'epoch': 0.01}                      \r\n",
      "{'loss': 0.0377, 'learning_rate': 1.7533333333333337e-05, 'epoch': 0.01}        \r\n",
      "{'loss': 0.8701, 'learning_rate': 1.7466666666666667e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.1365, 'learning_rate': 1.7400000000000003e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.763, 'learning_rate': 1.7333333333333336e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.6814, 'learning_rate': 1.726666666666667e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.066, 'learning_rate': 1.72e-05, 'epoch': 0.02}                       \r\n",
      "{'loss': 0.543, 'learning_rate': 1.7133333333333334e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.1186, 'learning_rate': 1.706666666666667e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.0753, 'learning_rate': 1.7e-05, 'epoch': 0.02}                       \r\n",
      "{'loss': 0.1376, 'learning_rate': 1.6933333333333336e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.0322, 'learning_rate': 1.686666666666667e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.2514, 'learning_rate': 1.6800000000000002e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.9785, 'learning_rate': 1.6733333333333335e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.4355, 'learning_rate': 1.6666666666666667e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.064, 'learning_rate': 1.66e-05, 'epoch': 0.02}                       \r\n",
      "{'loss': 0.0915, 'learning_rate': 1.6533333333333333e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.0999, 'learning_rate': 1.646666666666667e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.0058, 'learning_rate': 1.64e-05, 'epoch': 0.02}                      \r\n",
      "{'loss': 0.3339, 'learning_rate': 1.6333333333333335e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.604, 'learning_rate': 1.6266666666666668e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.3101, 'learning_rate': 1.62e-05, 'epoch': 0.02}                      \r\n",
      "{'loss': 0.149, 'learning_rate': 1.6133333333333334e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.1271, 'learning_rate': 1.606666666666667e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.2854, 'learning_rate': 1.6000000000000003e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.3322, 'learning_rate': 1.5933333333333336e-05, 'epoch': 0.02}        \r\n",
      "{'loss': 0.0556, 'learning_rate': 1.586666666666667e-05, 'epoch': 0.02}         \r\n",
      "{'loss': 0.1277, 'learning_rate': 1.58e-05, 'epoch': 0.03}                      \r\n",
      "{'loss': 1.4146, 'learning_rate': 1.5733333333333334e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0079, 'learning_rate': 1.5666666666666667e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0576, 'learning_rate': 1.5600000000000003e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.049, 'learning_rate': 1.5533333333333333e-05, 'epoch': 0.03}         \r\n",
      "{'loss': 0.4167, 'learning_rate': 1.546666666666667e-05, 'epoch': 0.03}         \r\n",
      "{'loss': 0.1903, 'learning_rate': 1.54e-05, 'epoch': 0.03}                      \r\n",
      "{'loss': 0.0972, 'learning_rate': 1.5333333333333334e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.056, 'learning_rate': 1.5266666666666667e-05, 'epoch': 0.03}         \r\n",
      "{'loss': 0.0937, 'learning_rate': 1.5200000000000002e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0168, 'learning_rate': 1.5133333333333335e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.4977, 'learning_rate': 1.5066666666666668e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.7477, 'learning_rate': 1.5000000000000002e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0054, 'learning_rate': 1.4933333333333335e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0664, 'learning_rate': 1.4866666666666668e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.1106, 'learning_rate': 1.48e-05, 'epoch': 0.03}                      \r\n",
      "{'loss': 0.3886, 'learning_rate': 1.4733333333333335e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0432, 'learning_rate': 1.4666666666666666e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.9208, 'learning_rate': 1.46e-05, 'epoch': 0.03}                      \r\n",
      "{'loss': 0.047, 'learning_rate': 1.4533333333333335e-05, 'epoch': 0.03}         \r\n",
      "{'loss': 0.184, 'learning_rate': 1.4466666666666668e-05, 'epoch': 0.03}         \r\n",
      "{'loss': 0.0166, 'learning_rate': 1.4400000000000001e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0051, 'learning_rate': 1.4333333333333334e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0221, 'learning_rate': 1.4266666666666668e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0343, 'learning_rate': 1.4200000000000001e-05, 'epoch': 0.03}        \r\n",
      "{'loss': 0.0725, 'learning_rate': 1.4133333333333334e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.612, 'learning_rate': 1.4066666666666669e-05, 'epoch': 0.04}         \r\n",
      "{'loss': 0.0592, 'learning_rate': 1.4e-05, 'epoch': 0.04}                       \r\n",
      "{'loss': 0.3324, 'learning_rate': 1.3933333333333334e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.4796, 'learning_rate': 1.3866666666666669e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0342, 'learning_rate': 1.38e-05, 'epoch': 0.04}                      \r\n",
      "{'loss': 0.1784, 'learning_rate': 1.3733333333333335e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.007, 'learning_rate': 1.3666666666666667e-05, 'epoch': 0.04}         \r\n",
      "{'loss': 0.1859, 'learning_rate': 1.3600000000000002e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.1524, 'learning_rate': 1.3533333333333333e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0107, 'learning_rate': 1.3466666666666668e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.2768, 'learning_rate': 1.3400000000000002e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0235, 'learning_rate': 1.3333333333333333e-05, 'epoch': 0.04}        \r\n",
      " 33%|█████████████▋                           | 100/300 [02:29<04:59,  1.50s/it][INFO|tokenization_utils_base.py:2428] 2023-11-24 13:46:55,413 >> tokenizer config file saved in output/news-20231124-134412-2e-05/checkpoint-100/tokenizer_config.json\r\n",
      "[INFO|tokenization_utils_base.py:2437] 2023-11-24 13:46:55,413 >> Special tokens file saved in output/news-20231124-134412-2e-05/checkpoint-100/special_tokens_map.json\r\n",
      "{'loss': 0.0081, 'learning_rate': 1.3266666666666668e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0147, 'learning_rate': 1.3200000000000002e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0156, 'learning_rate': 1.3133333333333334e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0188, 'learning_rate': 1.3066666666666668e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.1504, 'learning_rate': 1.3000000000000001e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.1548, 'learning_rate': 1.2933333333333334e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.1723, 'learning_rate': 1.2866666666666667e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0258, 'learning_rate': 1.2800000000000001e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0329, 'learning_rate': 1.2733333333333336e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.0042, 'learning_rate': 1.2666666666666667e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.2594, 'learning_rate': 1.2600000000000001e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.3348, 'learning_rate': 1.2533333333333336e-05, 'epoch': 0.04}        \r\n",
      "{'loss': 0.1135, 'learning_rate': 1.2466666666666667e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0225, 'learning_rate': 1.2400000000000002e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0036, 'learning_rate': 1.2333333333333334e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0105, 'learning_rate': 1.2266666666666667e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0565, 'learning_rate': 1.22e-05, 'epoch': 0.05}                      \r\n",
      "{'loss': 0.0286, 'learning_rate': 1.2133333333333335e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.5644, 'learning_rate': 1.206666666666667e-05, 'epoch': 0.05}         \r\n",
      "{'loss': 0.3605, 'learning_rate': 1.2e-05, 'epoch': 0.05}                       \r\n",
      "{'loss': 0.6441, 'learning_rate': 1.1933333333333335e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0111, 'learning_rate': 1.186666666666667e-05, 'epoch': 0.05}         \r\n",
      "{'loss': 0.0098, 'learning_rate': 1.18e-05, 'epoch': 0.05}                      \r\n",
      "{'loss': 0.9559, 'learning_rate': 1.1733333333333335e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0042, 'learning_rate': 1.1666666666666668e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.4136, 'learning_rate': 1.16e-05, 'epoch': 0.05}                      \r\n",
      "{'loss': 0.5268, 'learning_rate': 1.1533333333333334e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.1325, 'learning_rate': 1.1466666666666668e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0051, 'learning_rate': 1.14e-05, 'epoch': 0.05}                      \r\n",
      "{'loss': 0.0006, 'learning_rate': 1.1333333333333334e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.007, 'learning_rate': 1.1266666666666668e-05, 'epoch': 0.05}         \r\n",
      "{'loss': 0.441, 'learning_rate': 1.1200000000000001e-05, 'epoch': 0.05}         \r\n",
      "{'loss': 0.0102, 'learning_rate': 1.1133333333333334e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0415, 'learning_rate': 1.1066666666666669e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.0206, 'learning_rate': 1.1000000000000001e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.1697, 'learning_rate': 1.0933333333333334e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.2054, 'learning_rate': 1.0866666666666667e-05, 'epoch': 0.05}        \r\n",
      "{'loss': 0.005, 'learning_rate': 1.0800000000000002e-05, 'epoch': 0.06}         \r\n",
      "{'loss': 0.9604, 'learning_rate': 1.0733333333333333e-05, 'epoch': 0.06}        \r\n",
      "{'loss': 0.0043, 'learning_rate': 1.0666666666666667e-05, 'epoch': 0.06}        \r\n",
      "{'loss': 0.4815, 'learning_rate': 1.0600000000000002e-05, 'epoch': 0.06}        \r\n",
      "{'loss': 0.4817, 'learning_rate': 1.0533333333333333e-05, 'epoch': 0.06}        \r\n",
      "{'loss': 1.0441, 'learning_rate': 1.0466666666666668e-05, 'epoch': 0.06}        \r\n",
      "{'loss': 1.2198, 'learning_rate': 1.04e-05, 'epoch': 0.06}                      \r\n",
      "{'loss': 0.019, 'learning_rate': 1.0333333333333335e-05, 'epoch': 0.06}         \r\n",
      "{'loss': 0.0127, 'learning_rate': 1.0266666666666668e-05, 'epoch': 0.06}        \r\n",
      "{'loss': 0.0606, 'learning_rate': 1.02e-05, 'epoch': 0.06}                      \r\n",
      "{'loss': 0.1299, 'learning_rate': 1.0133333333333335e-05, 'epoch': 0.06}        \r\n",
      "{'loss': 0.0678, 'learning_rate': 1.0066666666666666e-05, 'epoch': 0.06}        \r\n",
      "{'loss': 0.0037, 'learning_rate': 1e-05, 'epoch': 0.06}                         \r\n",
      "{'loss': 0.4082, 'learning_rate': 9.933333333333334e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.1784, 'learning_rate': 9.866666666666668e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.0074, 'learning_rate': 9.800000000000001e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.148, 'learning_rate': 9.733333333333334e-06, 'epoch': 0.06}          \r\n",
      "{'loss': 0.0005, 'learning_rate': 9.666666666666667e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.3039, 'learning_rate': 9.600000000000001e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.0024, 'learning_rate': 9.533333333333334e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.2657, 'learning_rate': 9.466666666666667e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.0117, 'learning_rate': 9.4e-06, 'epoch': 0.06}                       \r\n",
      "{'loss': 0.001, 'learning_rate': 9.333333333333334e-06, 'epoch': 0.06}          \r\n",
      "{'loss': 0.0045, 'learning_rate': 9.266666666666667e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.0013, 'learning_rate': 9.200000000000002e-06, 'epoch': 0.06}         \r\n",
      "{'loss': 0.4838, 'learning_rate': 9.133333333333335e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.164, 'learning_rate': 9.066666666666667e-06, 'epoch': 0.07}          \r\n",
      "{'loss': 0.0084, 'learning_rate': 9e-06, 'epoch': 0.07}                         \r\n",
      "{'loss': 0.0207, 'learning_rate': 8.933333333333333e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.3102, 'learning_rate': 8.866666666666668e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0723, 'learning_rate': 8.8e-06, 'epoch': 0.07}                       \r\n",
      "{'loss': 0.0009, 'learning_rate': 8.733333333333333e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.1778, 'learning_rate': 8.666666666666668e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 1.0444, 'learning_rate': 8.6e-06, 'epoch': 0.07}                       \r\n",
      "{'loss': 0.1915, 'learning_rate': 8.533333333333335e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0191, 'learning_rate': 8.466666666666668e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0044, 'learning_rate': 8.400000000000001e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.5965, 'learning_rate': 8.333333333333334e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0201, 'learning_rate': 8.266666666666667e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 1.1224, 'learning_rate': 8.2e-06, 'epoch': 0.07}                       \r\n",
      "{'loss': 0.0041, 'learning_rate': 8.133333333333334e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0071, 'learning_rate': 8.066666666666667e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0885, 'learning_rate': 8.000000000000001e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.394, 'learning_rate': 7.933333333333334e-06, 'epoch': 0.07}          \r\n",
      "{'loss': 0.0002, 'learning_rate': 7.866666666666667e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.2249, 'learning_rate': 7.800000000000002e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.2116, 'learning_rate': 7.733333333333334e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0159, 'learning_rate': 7.666666666666667e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0027, 'learning_rate': 7.600000000000001e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.1615, 'learning_rate': 7.533333333333334e-06, 'epoch': 0.07}         \r\n",
      "{'loss': 0.0401, 'learning_rate': 7.4666666666666675e-06, 'epoch': 0.08}        \r\n",
      "{'loss': 0.0089, 'learning_rate': 7.4e-06, 'epoch': 0.08}                       \r\n",
      "{'loss': 0.0712, 'learning_rate': 7.333333333333333e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.5022, 'learning_rate': 7.266666666666668e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.1132, 'learning_rate': 7.2000000000000005e-06, 'epoch': 0.08}        \r\n",
      "{'loss': 0.3318, 'learning_rate': 7.133333333333334e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.0071, 'learning_rate': 7.066666666666667e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.6094, 'learning_rate': 7e-06, 'epoch': 0.08}                         \r\n",
      "{'loss': 0.0022, 'learning_rate': 6.9333333333333344e-06, 'epoch': 0.08}        \r\n",
      "{'loss': 0.4046, 'learning_rate': 6.866666666666667e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.3455, 'learning_rate': 6.800000000000001e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.5033, 'learning_rate': 6.733333333333334e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.1898, 'learning_rate': 6.666666666666667e-06, 'epoch': 0.08}         \r\n",
      " 67%|███████████████████████████▎             | 200/300 [05:00<02:30,  1.51s/it][INFO|tokenization_utils_base.py:2428] 2023-11-24 13:49:25,887 >> tokenizer config file saved in output/news-20231124-134412-2e-05/checkpoint-200/tokenizer_config.json\r\n",
      "[INFO|tokenization_utils_base.py:2437] 2023-11-24 13:49:25,887 >> Special tokens file saved in output/news-20231124-134412-2e-05/checkpoint-200/special_tokens_map.json\r\n",
      "{'loss': 0.2219, 'learning_rate': 6.600000000000001e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.0261, 'learning_rate': 6.533333333333334e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.0268, 'learning_rate': 6.466666666666667e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.2255, 'learning_rate': 6.4000000000000006e-06, 'epoch': 0.08}        \r\n",
      "{'loss': 0.0023, 'learning_rate': 6.333333333333333e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.003, 'learning_rate': 6.266666666666668e-06, 'epoch': 0.08}          \r\n",
      "{'loss': 0.0272, 'learning_rate': 6.200000000000001e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.0585, 'learning_rate': 6.133333333333334e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.0094, 'learning_rate': 6.066666666666667e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.163, 'learning_rate': 6e-06, 'epoch': 0.08}                          \r\n",
      "{'loss': 0.0036, 'learning_rate': 5.933333333333335e-06, 'epoch': 0.08}         \r\n",
      "{'loss': 0.0781, 'learning_rate': 5.8666666666666675e-06, 'epoch': 0.08}        \r\n",
      "{'loss': 0.0089, 'learning_rate': 5.8e-06, 'epoch': 0.09}                       \r\n",
      "{'loss': 0.5827, 'learning_rate': 5.733333333333334e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.0053, 'learning_rate': 5.666666666666667e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.0179, 'learning_rate': 5.600000000000001e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 1.8615, 'learning_rate': 5.533333333333334e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.3423, 'learning_rate': 5.466666666666667e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.5502, 'learning_rate': 5.400000000000001e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.1772, 'learning_rate': 5.333333333333334e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.075, 'learning_rate': 5.2666666666666665e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.0059, 'learning_rate': 5.2e-06, 'epoch': 0.09}                       \r\n",
      "{'loss': 0.0042, 'learning_rate': 5.133333333333334e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.0213, 'learning_rate': 5.0666666666666676e-06, 'epoch': 0.09}        \r\n",
      "{'loss': 0.0174, 'learning_rate': 5e-06, 'epoch': 0.09}                         \r\n",
      "{'loss': 1.0351, 'learning_rate': 4.933333333333334e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.0026, 'learning_rate': 4.866666666666667e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.0026, 'learning_rate': 4.800000000000001e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.5251, 'learning_rate': 4.7333333333333335e-06, 'epoch': 0.09}        \r\n",
      "{'loss': 0.2601, 'learning_rate': 4.666666666666667e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.006, 'learning_rate': 4.600000000000001e-06, 'epoch': 0.09}          \r\n",
      "{'loss': 0.8758, 'learning_rate': 4.533333333333334e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.2939, 'learning_rate': 4.4666666666666665e-06, 'epoch': 0.09}        \r\n",
      "{'loss': 0.0141, 'learning_rate': 4.4e-06, 'epoch': 0.09}                       \r\n",
      "{'loss': 0.001, 'learning_rate': 4.333333333333334e-06, 'epoch': 0.09}          \r\n",
      "{'loss': 0.0176, 'learning_rate': 4.266666666666668e-06, 'epoch': 0.09}         \r\n",
      "{'loss': 0.4934, 'learning_rate': 4.2000000000000004e-06, 'epoch': 0.09}        \r\n",
      "{'loss': 0.3413, 'learning_rate': 4.133333333333333e-06, 'epoch': 0.1}          \r\n",
      "{'loss': 0.0059, 'learning_rate': 4.066666666666667e-06, 'epoch': 0.1}          \r\n",
      "{'loss': 0.0379, 'learning_rate': 4.000000000000001e-06, 'epoch': 0.1}          \r\n",
      "{'loss': 0.7871, 'learning_rate': 3.9333333333333335e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.0035, 'learning_rate': 3.866666666666667e-06, 'epoch': 0.1}          \r\n",
      "{'loss': 0.3481, 'learning_rate': 3.8000000000000005e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.1827, 'learning_rate': 3.7333333333333337e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.4669, 'learning_rate': 3.6666666666666666e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.8078, 'learning_rate': 3.6000000000000003e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 1.1314, 'learning_rate': 3.5333333333333335e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.0124, 'learning_rate': 3.4666666666666672e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.0116, 'learning_rate': 3.4000000000000005e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.2296, 'learning_rate': 3.3333333333333333e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.0037, 'learning_rate': 3.266666666666667e-06, 'epoch': 0.1}          \r\n",
      "{'loss': 0.0882, 'learning_rate': 3.2000000000000003e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.0073, 'learning_rate': 3.133333333333334e-06, 'epoch': 0.1}          \r\n",
      "{'loss': 0.013, 'learning_rate': 3.066666666666667e-06, 'epoch': 0.1}           \r\n",
      "{'loss': 0.0027, 'learning_rate': 3e-06, 'epoch': 0.1}                          \r\n",
      "{'loss': 0.7103, 'learning_rate': 2.9333333333333338e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.0022, 'learning_rate': 2.866666666666667e-06, 'epoch': 0.1}          \r\n",
      "{'loss': 0.4343, 'learning_rate': 2.8000000000000003e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 1.1652, 'learning_rate': 2.7333333333333336e-06, 'epoch': 0.1}         \r\n",
      "{'loss': 0.377, 'learning_rate': 2.666666666666667e-06, 'epoch': 0.1}           \r\n",
      "{'loss': 0.0025, 'learning_rate': 2.6e-06, 'epoch': 0.1}                        \r\n",
      "{'loss': 0.072, 'learning_rate': 2.5333333333333338e-06, 'epoch': 0.1}          \r\n",
      "{'loss': 0.0035, 'learning_rate': 2.466666666666667e-06, 'epoch': 0.11}         \r\n",
      "{'loss': 0.0176, 'learning_rate': 2.4000000000000003e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0404, 'learning_rate': 2.3333333333333336e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0243, 'learning_rate': 2.266666666666667e-06, 'epoch': 0.11}         \r\n",
      "{'loss': 0.0197, 'learning_rate': 2.2e-06, 'epoch': 0.11}                       \r\n",
      "{'loss': 0.0558, 'learning_rate': 2.133333333333334e-06, 'epoch': 0.11}         \r\n",
      "{'loss': 0.0396, 'learning_rate': 2.0666666666666666e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0042, 'learning_rate': 2.0000000000000003e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.2281, 'learning_rate': 1.9333333333333336e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0594, 'learning_rate': 1.8666666666666669e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0016, 'learning_rate': 1.8000000000000001e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.1489, 'learning_rate': 1.7333333333333336e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0053, 'learning_rate': 1.6666666666666667e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0131, 'learning_rate': 1.6000000000000001e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0517, 'learning_rate': 1.5333333333333334e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 2.0054, 'learning_rate': 1.4666666666666669e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.863, 'learning_rate': 1.4000000000000001e-06, 'epoch': 0.11}         \r\n",
      "{'loss': 0.0038, 'learning_rate': 1.3333333333333334e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0003, 'learning_rate': 1.2666666666666669e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0553, 'learning_rate': 1.2000000000000002e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0076, 'learning_rate': 1.1333333333333334e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.0481, 'learning_rate': 1.066666666666667e-06, 'epoch': 0.11}         \r\n",
      "{'loss': 0.5921, 'learning_rate': 1.0000000000000002e-06, 'epoch': 0.11}        \r\n",
      "{'loss': 0.3559, 'learning_rate': 9.333333333333334e-07, 'epoch': 0.11}         \r\n",
      "{'loss': 0.0014, 'learning_rate': 8.666666666666668e-07, 'epoch': 0.11}         \r\n",
      "{'loss': 0.4355, 'learning_rate': 8.000000000000001e-07, 'epoch': 0.12}         \r\n",
      "{'loss': 0.0764, 'learning_rate': 7.333333333333334e-07, 'epoch': 0.12}         \r\n",
      "{'loss': 0.0035, 'learning_rate': 6.666666666666667e-07, 'epoch': 0.12}         \r\n",
      "{'loss': 0.0042, 'learning_rate': 6.000000000000001e-07, 'epoch': 0.12}         \r\n",
      "{'loss': 0.0086, 'learning_rate': 5.333333333333335e-07, 'epoch': 0.12}         \r\n",
      "{'loss': 0.2114, 'learning_rate': 4.666666666666667e-07, 'epoch': 0.12}         \r\n",
      "{'loss': 0.223, 'learning_rate': 4.0000000000000003e-07, 'epoch': 0.12}         \r\n",
      "{'loss': 0.0044, 'learning_rate': 3.3333333333333335e-07, 'epoch': 0.12}        \r\n",
      "{'loss': 0.6593, 'learning_rate': 2.666666666666667e-07, 'epoch': 0.12}         \r\n",
      "{'loss': 0.1374, 'learning_rate': 2.0000000000000002e-07, 'epoch': 0.12}        \r\n",
      "{'loss': 0.1037, 'learning_rate': 1.3333333333333336e-07, 'epoch': 0.12}        \r\n",
      "{'loss': 0.0181, 'learning_rate': 6.666666666666668e-08, 'epoch': 0.12}         \r\n",
      "{'loss': 0.0783, 'learning_rate': 0.0, 'epoch': 0.12}                           \r\n",
      "100%|█████████████████████████████████████████| 300/300 [07:31<00:00,  1.51s/it][INFO|tokenization_utils_base.py:2428] 2023-11-24 13:51:56,849 >> tokenizer config file saved in output/news-20231124-134412-2e-05/checkpoint-300/tokenizer_config.json\r\n",
      "[INFO|tokenization_utils_base.py:2437] 2023-11-24 13:51:56,849 >> Special tokens file saved in output/news-20231124-134412-2e-05/checkpoint-300/special_tokens_map.json\r\n",
      "[INFO|trainer.py:1955] 2023-11-24 13:51:56,875 >> \r\n",
      "\r\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\r\n",
      "\r\n",
      "\r\n",
      "{'train_runtime': 451.0866, 'train_samples_per_second': 1.33, 'train_steps_per_second': 0.665, 'train_loss': 0.2702767427762349, 'epoch': 0.12}\r\n",
      "100%|█████████████████████████████████████████| 300/300 [07:31<00:00,  1.50s/it]\r\n",
      "[INFO|tokenization_utils_base.py:2428] 2023-11-24 13:51:56,887 >> tokenizer config file saved in output/news-20231124-134412-2e-05/tokenizer_config.json\r\n",
      "[INFO|tokenization_utils_base.py:2437] 2023-11-24 13:51:56,887 >> Special tokens file saved in output/news-20231124-134412-2e-05/special_tokens_map.json\r\n"
     ]
    }
   ],
   "source": [
    "!which python\n",
    "import os\n",
    "from datetime import datetime\n",
    "import random\n",
    "\n",
    "# 定义变量\n",
    "lr = 2e-5\n",
    "num_gpus = 1\n",
    "lora_rank = 8\n",
    "lora_alpha = 32\n",
    "lora_dropout = 0.1\n",
    "max_source_len = 512\n",
    "max_target_len = 128\n",
    "dev_batch_size = 1\n",
    "grad_accumularion_steps = 2\n",
    "max_step = 300\n",
    "save_interval = 100\n",
    "max_seq_len = 512\n",
    "logging_steps=1\n",
    "\n",
    "run_name = \"news\"\n",
    "dataset_path = \"data/toutiao_cat_data_example.jsonl\"\n",
    "datestr = datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
    "output_dir = f\"output/{run_name}-{datestr}-{lr}\"\n",
    "master_port = random.randint(10000, 65535)\n",
    "\n",
    "os.makedirs(output_dir, exist_ok=True)\n",
    "# 构建命令\n",
    "command = f\"\"\"\n",
    "torchrun --standalone --nnodes=1 --nproc_per_node={num_gpus} ../finetune_basemodel_demo/finetune.py \\\n",
    "    --train_format input-output \\\n",
    "    --train_file {dataset_path} \\\n",
    "    --lora_rank {lora_rank} \\\n",
    "    --lora_alpha {lora_alpha} \\\n",
    "    --lora_dropout {lora_dropout} \\\n",
    "    --max_seq_length {max_seq_len} \\\n",
    "    --preprocessing_num_workers 1 \\\n",
    "    --model_name_or_path {model_path} \\\n",
    "    --output_dir {output_dir} \\\n",
    "    --per_device_train_batch_size 1 \\\n",
    "    --gradient_accumulation_steps 2 \\\n",
    "    --max_steps {max_step} \\\n",
    "    --logging_steps {logging_steps} \\\n",
    "    --save_steps {save_interval} \\\n",
    "    --learning_rate {lr}\n",
    "\"\"\"\n",
    "\n",
    "# 在 Notebook 中执行命令\n",
    "!{command}"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:44:12.468751Z",
     "end_time": "2023-11-24T13:51:59.534815Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 使用微调的模型进行推理预测\n",
    "现在，我们已经完成了模型的微调，接下来，我们将使用微调后的模型进行推理。我们使用与微调时相同的提示词，并使用一些没有出现的模型效果来复现推理结果。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "data": {
      "text/plain": "Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "21b29455e65c4540886188d1fd5d68aa"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoModel, AutoTokenizer\n",
    "import os\n",
    "from peft import get_peft_model, LoraConfig, TaskType\n",
    "\n",
    "# 参数设置\n",
    "lora_path = output_dir + \"pytorch_model.bin\"\n",
    "\n",
    "# 加载分词器和模型\n",
    "tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, trust_remote_code=True)\n",
    "model = AutoModel.from_pretrained(model_path, load_in_8bit=False, trust_remote_code=True).to(device)\n",
    "\n",
    "# LoRA 模型配置\n",
    "peft_config = LoraConfig(\n",
    "    task_type=TaskType.CAUSAL_LM, inference_mode=True,\n",
    "    target_modules=['query_key_value'],\n",
    "    r=8, lora_alpha=32, lora_dropout=0.1\n",
    ")\n",
    "model = get_peft_model(model, peft_config)\n",
    "if os.path.exists(lora_path):\n",
    "    model.load_state_dict(torch.load(lora_path), strict=False)\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:51:59.522046Z",
     "end_time": "2023-11-24T13:52:12.130299Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "我们使用同样的提示词进行推理。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [],
   "source": [
    "inputs = tokenizer(PROMPT, return_tensors=\"pt\").to(device)\n",
    "response = model.generate(input_ids=inputs[\"input_ids\"],max_new_tokens=max_new_tokens,temperature=temperature,top_p=top_p,do_sample=True)\n",
    "response = response[0, inputs[\"input_ids\"].shape[-1]:]\n",
    "response = tokenizer.decode(response, skip_special_tokens=True)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:52:12.132308Z",
     "end_time": "2023-11-24T13:52:12.296837Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [
    {
     "data": {
      "text/plain": "'news_tech'"
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:52:12.296837Z",
     "end_time": "2023-11-24T13:52:12.297910Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "这一次，模型成功给出了理想的答案。我们结束实操训练，删除模型并释放显存。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [],
   "source": [
    "del model\n",
    "torch.cuda.empty_cache()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-11-24T13:54:07.355796Z",
     "end_time": "2023-11-24T13:54:07.507254Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 总结\n",
    "在本实践手册中，我们让开发者体验了 `ChatGLM3-6B` 模型在经过微调前后在 `新闻标题分类` 任务中表现。\n",
    "我们可以发现：\n",
    "\n",
    "在具有混淆的新闻分类中，原始的模型可能受到了误导，不能有效的进行分类，而经过简单微调后的模型，已经具备了正确分类的能力。\n",
    "因此，对于有更高要求的专业分类任务，我们可以使用微调的方式对模型进行简单微调，实现更好的任务完成效果。\n"
   ],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
